Method of maintaining accuracy in a 3D image formation system

ABSTRACT

A method and system for a displaying content that makes LED real-time 3D displays possible. The 3D image formation includes multi-screen display color correction and multi-camera infrared positioning technology. The present invention presents the display effects of a simulation environment in a better way through switching and moving and ensures that while in movement, the display effects of LED screens taken by video cameras from different angles can be maintained the same, so as to create a vivid simulation environment for television programming and film shooting. The system for maintaining accuracy in a 3D image formation includes a multi-screen display color correction system, a multi-screen display color correction system comprising a large-size screen server, a video camera and a large-size display screen wherein the large-size screen color correction server includes an image input module, a position data module, a comprehensive color correction server and an image output module.

PRIORITY AND RELATED APPLICATIONS

This application is a Divisional of U.S. patent application Ser. No.15/719,772 filed Sep. 29, 2017 entitled “THREE-DIMENSIONAL IMAGEFORMATION AND COLOR CORRECTION SYSTEM AND METHOD,” which is herebyincorporated by reference in its entirety.

FIELD OF TECHNOLOGY

This invention relates to a 3D image-formation system, more specificallya system and method used for LED display including multi-screen displaycolor correction and multi camera infrared positioning technology.

BACKGROUND

LED large-size screens are extensively utilized in programming, displayand advertising. But when utilized, multiple LED screens can havedifferent in viewing angles, positions and directions, and style and dotspacing. When using a video camera to shoot, there will be chromaticaberration, leading to the failed integration of the viewing displayeffect with the display effect after shooting, as they have evidentdifferences in boundary lines and brightness and darkness. Not all thescreens can have the same shooting effects. And the color adjustment ofthe current LED screens in the market uses a pre-set fixed value, whichcannot be matched or adjusted dynamically while in the shooting process.

An LED display is realized by the permutation and combination of minorLED lights of the three basic colors (red, green and blue). So indifferent viewing angles, there will be different chromatic aberrationdisplays, having great influence on viewing and shooting screen effects.When shooting LED screens, normal colors and color temperature can beshown only when the video camera is facing the screen directly. But whenmultiple screens are joined together, it's unable to ensure the videocamera is facing all the screens directly or shooting from the one fixedunified angle, so the display effect of LED screens is greatlydestroyed.

Currently, video cameras have multiple methods of positioning, includinginfrared positioning, visual positioning, laser positioning, ultrasonicpositioning and so on. But they don't have a position tracking system toconduct color correction on the screen display.

The multi-camera infrared positioning technology is often used in filmand television, including skeletal animation, positioning of the videocamera and moving track, moving coordinates of relevant marked goods andso on. However, there is no positioning application method ofmulti-image space matched with the 3D integration of severalmultifaceted irregular screens.

In a known layout, a little infrared emitter is installed on the head ofthe several video cameras and one to two infrared sensors are addedabove the space for shooting.

The two infrared sensors 315 in the prior art measure the distancebetween the video camera 330 and the two infrared sensors namelydistance L1 and distance L2 by receiving the infrared signal sent by theinfrared emitter on the head of the video camera. Then the position ofthe cradle head, as shown in FIG. 4 can be calculated. Taking the twoinfrared sensors 315 as the centers, two circles C1, C2 with radius ofdistance L1 and distance L2 can be drawn respectively. The intersectionpoint 410 is the position of the video camera. Using the positions ofthe video camera in the space can be determined respectively.Accordingly, calculations using the position of the video camera areused to later form the image. When switching the video cameras, theimages taken by them are matched with their real positions.

Current multi-camera infrared positioning technology gets the neededcoordinate information only, by locating the same space position andcannot process the positioning data of multi-virtual space featured asmulti-angle and multifaceted and synchronize the coordinate relationshipof different virtual spaces. Currently, there are no switchable methodsof multiple position coordinates that can be applied to multi-screendisplay 3D space, and therefore fail to satisfy the applications of the3D integration space of multifaceted irregular screens.

Some major 3D image-formation technologies in current technologiesinclude a LED display technology and Space Coordinate MatchingTechnology. In LED screen display technology the current display methodis passive and flat. No matter how the LED screens are pieced together,they cannot display an accurate 3D image. Furthermore, the displayedimage will not move according to the position of the viewer in real timeand cannot stimulate an accurate 3D space environment. Large-size LEDscreens have been used as display units or mediums. In the LED screenindustry, there is no relevant positive screen display technology.

Space coordinate matching technology focuses on virtual realityapplications. While there are a few applications introducing coordinatesin a virtual world matching real environment there are no applicationswith multi-things and multi-viewpoints introducing coordinates in avirtual world matching real environment.

Chinese Publication CN103941851A teaches a method and system ofrealizing virtual touch calibration. The method includes creating avirtual calibration list; building up a 1^(st) coordinate system basedon the surface located by the virtual calibration list; building up a2^(nd) coordinate system to present the gesture positions of the users;calculating the correspondence between the above-mentioned 1^(st)coordinate system and 2^(nd) coordinate system; based on thecorrespondence, using the coordinate of the 1^(st) coordinate system toshow the gesture positions of the users presented by the 2^(nd)coordinate system; using the gesture positions of the users shown by thecoordinate of the 1^(st) coordinate system to calibrate thecorrespondence between the users' gestures and the virtual calibrationlist.

In known technologies, 3D virtual projection and touching userinterfaces in virtual environments and its implementation methods caninclude one or more of the depth detecting device, the parallax of abinocular calculating module, the binocular image processing module, the3D display device, the gesture recognition module, and the camera andthe touch controller in a virtual environment.

Chinese Publication CN103941851A teaches recalibrating the users gesturewhen the position of the depth detecting device is changed or thedistance of pupils is changed after the change of users. The technologyof clicking the calibration point on the virtual calibration list by theusers will be adopted to re-calibrate users' gesture operation andvirtual projection images, so as to effectively solve the inconformitybetween click of gesture and response when the above-mentioned changehappens in the current technologies to maintain the accuracy of theinteraction.

Chinese Publication CN103365572 teaches a long-distance control methodof an electronic device. This method is applied between a 1^(st)electronic device and the 2^(nd) electronic device, which are connectedto each other by wireless ways. The 1^(st) electronic device includes animage acquisition device and a touch control display unit; the 2^(nd)electronic device contains a display unit. The method includes: the1^(st) electronic device acquires real-time images of the 1^(st) displaycontents displayed by the above-mentioned display unit by theabove-mentioned images and display the real-time images in the touchdisplay control unit; building up the relationship of coordinatetransformation between corresponding display coordinates of thereal-time images and the corresponding display coordinate of the 1^(st)display contents; and detecting the touch control operation informationreceived by the touch display unit and making sure whether the touchpoint coordinate of the touch control operation corresponds with the1^(st) display contents in the real-time images. If yes, transformingthe corresponding touch control point coordinate of touch controloperation into a 2^(nd) coordinate in the display unit according to the1^(st) display coordinate transform relationship and sending the touchcontrol directive in the touch control operation information to the2^(nd) electronic device, making it realize the operation on the 2^(nd)coordinate position through the touch control directive.

Chinese Publication 103365572 provides the ability to use an electronicdevice with a zoom camera and touch screen to control another electronicdevice. When users gain the displayed contents in another electronicdevice display unit through the camera, it will be displayed on a touchscreen. Users can control the desktop of another electronic devicewithin a certain distance by using the touch screen. The control of anon-touch control display screen can be achieved through by way oftouching control.

Chinese Publication CN105159522A teaches a virtual reality displaydevice to respond to peripheral devices. Specific methods include thevirtual reality display device having two display screens. Each of thedisplay screens corresponds with a part of the whole interaction scope.

The method includes: gaining the current position coordinate of aperipheral device; transforming the current position coordinate with acorresponding method of the pre-set conditions and getting theresponding position coordinate in the designated area. The designatedarea refers to the corresponding interaction scope of the designateddisplay screen between the two display screens; and conducting positioninteraction based on the coordinate of the responding position.

Chinese Publication CN105159522 teaches getting a responding positioncoordinate in a designated scope by transforming a current positioncoordinate through an attached peripheral device, so as to make thevirtual reality display device able to respond to the operation of theperipheral device and realize the interaction with the respondingposition coordinate of the peripheral device. The responding positioncoordinate after being transformed is limited to the designated scope,being able to prevent when the current position coordinate interactswith the virtual reality display device, the responding positioncoordinate of the 2D input leaps into the 3D film and image, so as toovercome the uncomfortable feeling caused by this when users areexperiencing the virtual reality.

Chinese Publication CN102508543 teaches a user interface and arealization method of realizing 3D virtual projection and virtual touchin a display device. It includes the following components: a depthdetecting device, used for detecting the information of the distancebetween user's head and hands and the 3D display device; a Binocularimage optical parallax calculation module, the module calculates thebinocular image optical parallax between the 3D display virtualprojection and the scope of users' head and arm length of the userinterface according to the received distance information; binocularimage processing module, the module is for making images displayed bythe left and right eyes to reach the two-eye parallax image calculatedby the binocular image optical parallax calculation module and thensends the images after processing to the 3D display device; a 3D displaydevice for conducting 3D displays of the images for binocular processedby the binocular image processing module and using the user interface todisplay them within the scope of a user's head and arm lengths in theway of 3D virtual projection; a gesture recognition module whichcaptures user finger moving tracks through the camera and combines thedepth detecting device to gain information of the distance betweenusers' hands and the 3D display device and recognized hand gestures; acamera that captures user finger moving tracks; and a virtual touchcontroller for receiving the information from the gesture recognitionmodule and making a corresponding response. The output end of the depthdetecting device is connected with the input end of the parallax ofbinocular image optical parallax calculation module. The parallaxcalculation module is connected with the input end of the parallax ofbinocular image processing module, which is connected with the 3Ddisplay device. The input end of the gesture recognition module isconnected with the depth detecting device and the camera respectivelyand the output end of the gesture recognition module is connected withthe virtual touch controller.

Chinese Publication CN102508546 teaches utilizing depth detectiontechnology, 3D display technology and gesture recognition technology. Itcreates a brand-new 3D virtual touch interaction method to overcome thecurrent technology's problems that touching must stick to the screen andgestures need to be done within a certain distance from the interactivedevice. Users can not only conduct the touch operation on the virtualscreens, but also realize 3D virtual projection. This invention can notonly provide a 3D user interface with feedback, virtual projection andvirtual touch, but also brings a convenient and brand-new interactionexperience to users.

Chinese Publication CN103744518 teaches a 3D interaction method and adisplay device and system. This invention includes: conducting 3Dinteraction with the objects under operation displayed on the screen ofthe 3D display device through the 3D interaction operation stick;getting the position information of the viewer and conducting a 3Dadjustment display based on the parallax in the implementation processof the above-mentioned 3D interaction, according to the changingposition information. This publication allows users to view formerblocked images from other angles. In this process of the 3D interactionbetween the 3D interaction operation stick and the objects underoperation, when users' sight is blocked by the 3D interaction sticks,hands or other things, only by changing the view position, namelyadjusting the display effect of the screen based on the parallaxchanges, users are able to view the former blocked images from otherangles. It's convenient for users to do the 3D interaction operation ofthe objects under operation without suspending the operation.

SUMMARY OF THE INVENTION

An object of the present invention is to provide real-time colorcorrection software to set parameters, and output color parametersrelated to each screen in real time to adjust the display effect, so asto achieve a unified viewing effect from any angle, presenting overallyunified pictures no matter if it is for an audience to watch or forprogram shooting.

The method includes the following steps:

1) Gaining position coordinate information of a video camera and imagesof the multiple screens collected by the above-mentioned video camera.

2) Taking one of the above-mentioned multiple screens as the standardscreen and according to the collected images of the multiple screensconducting color correction on the display contents of the otherscreens, making the display of images collected by the video camera onthe other screens achieve the same display effect with that of thestandard screen and storing the screen correction parameters conductedto the other screens.

3) Making a judgment of the time of changing of the position of theabove-mentioned video camera. If the time is less than or equal to N,then changing the position of the video camera and returning to Step 1);if not, skipping to Step 4).

4) According to the stored multiple screen color correction parameters,using the matched curves of algorithms to guarantee the gain of screencolor correction parameters of the video camera in different positions,so as to make images of the several different screens collected by thevideo camera in different positions present the same effects.

5) When the video camera is in different positions, conducting colorcorrection on other screens according to the screen color correctionparameters gained by above-mentioned position coordinate information andthe matched curves.

The above-mentioned coordinate information at least includes: coordinateinformation and camera angle information of the video camera.

The above-mentioned standard screen is selected by the followingmethods: taking the screen that faces directly against the position ofthe video camera.

The above-mentioned N can be the value of 5, and can be adjusted tooptimize the overall display.

The above-mentioned same display effect refers to the condition whereinthe images of the several screens collected by the video camera indifferent positions and from different angles all present the same colorand color temperature, with no difference in colors.

The present invention provides a multi-screen display color correctionsystem, which includes: a large-size screen server, a video camera, alarge-size display screen, and a featured color correction server. Thecolor correction server includes an image input module, a position datamodule, a comprehensive color correction module and an image outputmodule. The comprehensive color correction module receives the videocamera position coordinate information transmitted by the position datamodule, and gains corresponding screen color correction parametersaccording to the position information of the video camera. Thenaccording to the parameters conducts color correction on the screendisplay contents sent by the image input module. The screen displaycontents are output after correction to the large-size display screen bythe image output module.

The large-size display screen includes multiple display screens.

The comprehensive color correction module gains the color correctionparameters of the multiple display screens according to the differentpositions of the video camera.

The video camera includes a video camera tracking sensor to collect thecoordinate information and camera angle information of the video camera.

The multiple display screens have the same display effect.

The present invention ensures that while in movement, the displayeffects of LED screens taken by the video camera from different anglescan be maintained the same, so as to create vivid simulation environmentfor television programming and film shooting.

Another objective of the present invention is to provide amulti-viewpoint switched shooting system and method based on an infraredpositioning system, used to conduct multi-angle and switchable shootingof the multifaceted irregular screen 3D integration space by acquiringthe position parameters of the infrared markers on the video camera andmatching the parameter information to different space coordinate systemsrespectively. Through the adjustment of algorithms, the presentinvention makes corresponding and synchronous corrections of differentspace coordinates and matches the coordinates with the positions of thevideo camera in a real environment. The coordinates of the several videocameras in the positioning system are marked and the coordinateinformation of the real and relevant video cameras is recognized whileswitching the video camera, making the 3D space image displayed by eachvideo camera angle match with the real positions.

The invention provides a multi-viewpoint switched shooting system basedon an infrared positioning system, which includes: a multifacetedirregular screen, a video splicer, a 3D rendering server, an infraredpositioning system, a motion-capture computer, a network switch and amultichannel control server.

The video splicer is installed in the back of the multifaceted irregularscreen. The 3D rendering server, the multichannel control server and thenetwork switch are connected with the multifaceted irregular screenrespectively and are used to process the images displayed on themultifaceted irregular screen. The infrared positioning system isinstalled above the multifaceted irregular screen and is connected withthe motion-capture computer to track the video camera shooting.

Based on the above schemes, the infrared positioning system includesseveral infrared cameras installed above the space for shooting; severalvideo cameras are placed in front of the space for shooting and each isequipped with several infrared markers on the head of the camera.

The infrared positioning system is used to get the position parametersof the infrared markers on the video camera.

The motion-capture computer is used to correct the different valuesbetween the infrared markers and the lens of the video camera based onthe position parameters of the infrared markers to get the coordinateinformation of the video camera and transmit it through the networkswitch to the 3D rendering server and the multichannel control server.

The 3D rendering server outputs the display of 3D space images.

The multichannel control server matches the coordinate information ofthe video camera to the virtual space for usage and matches thecoordinate information of the video camera with the coordinateinformation of the virtual space.

A multi-viewpoint switched shooting method based on the infraredpositioning system, includes the following steps:

Step 1, get the position parameters of the infrared markers on all thevideo cameras through the infrared positioning system and send theparameters to the motion-capture computer through the Network switch;

Step 2, use the motion-tracking software in the motion-capture computerto correct the difference value between the infrared markers and thelens of the video cameras according to the position parameters of theinfrared markers. Then transmit the parameter information to the 3Drendering server and the multichannel control server;

Step 3, project the parameter information of the video camera to thevirtual space via the multichannel control server. Through theadjustment of the image projection algorithm, match the parameterinformation of the video camera with the parameter information of thevideo camera in the virtual space;

Step 4, the 3D rendering server outputs the display of the 3D spaceimage according to the images taken by the video camera and theparameter information of the video camera in the virtual space;

Step 5, operate the motion-capture computer to switch the video camera.At the same time, the system switches the angle and position of virtualscenes, so as to make the displayed 3D space image match with the imagestaken by the video camera. The 3D space image presents a complete 3Dscene through the display of different screens.

The motion-tracking software used can be MotiveTracker and its plugins,for example.

The present invention uses four independent 3D rendering servers tooutput the display of 3D space images, the multi-camera infraredpositioning system to locate the position of the video camera and theinfrared markers to mark the video camera. The position parameters ofthe infrared markers of each video camera are recognized through theinfrared positioning system and the different values between theinfrared markers and the lens of the video camera are correctedrespectively. The corrected coordinate information is matched todifferent space coordinate systems. The different space coordinates arematched through the adjustment of algorithms. The coordinate informationof relevant video cameras are recognized while switching the videocamera, so as to make the displayed 3D space match with it.

The benefits of the present invention include:

1) This invention ensures that when different video cameras shootseveral LED screens with different angles from different angles, the 3Dspace can maintain a normal display and right display effect afterswitching the camera position without being stretched or distorted.

2) This invention provides better display effects of simulationenvironment space through multi-camera switching and moving. The variousshooting effects can be used for television program production and filmshooting.

3) This invention provides new shooting methods and expands new thinkingfor program creation for broadcast, television and film production. Itsupports the presentation of grand scenes, such as oceans, forests,aerospace and abstraction space, in very small space.

The present invention provides a LED 3D image-formation method,including the following steps:

Step 1, setting mesh for several LED displays in the virtual 3D space;

Step 2, Getting the position data of users in a real environment;

Step 3, transforming the above-mentioned position data into the positiondata in the virtual space;

Step 4, locating a user's position in the virtual 3D space based on theabove-mentioned position data;

Step 5, shooting in the virtual 3D space according to the location ofthe user in the virtual 3D space, using an orthographic camera;

Step 6, displaying the above-mentioned contents of virtual 3D space shotby the orthographic camera on several LED display screens.

The above-mentioned position data of the user in a real environment isthe coordinates of the user in the character position coordinate space,the above-mentioned virtual space position data is the coordinates ofthe user in the virtual space coordinate space.

The above-mentioned several LED display screens include at least: a LEDdisplay screen facing the user, a LED display screen in the left side ofthe user, a LED display screen in the right side of the user, and a LEDdisplay screen in the bottom of the user.

The contents in virtual space are adjusted and displayed on theabove-mentioned several LED display screens according to the position,area and quantity of the above-mentioned LED display screens.

A model of the virtual 3D space can be made through 3DMAX prior toStep 1) above, to set the mesh for the area and coordinate of theabove-mentioned several LED display screens.

The output contents shot by an orthographic camera include images and/orvideos.

The present invention provides a LED 3D image-formation system, whichincludes: a server, a positioning system, a large-size screen splicerand several LED display screens. The server is made up of a trackingmodule, a large-size screen display module and a virtual scene module.

A coordinate tracker tracks the position of users and the trackingsensor is used to receive the parameters of users' position given by thecoordinate tracker and sends the parameters to the server. The serverutilizes the tracking module to deal with the parameters and gets thecoordinate position data of users in a real environment, transformingthem into the virtual space position data in virtual 3D space. Thelarge-size screen display module outputs the virtual space contentsaccording to the virtual space position data, the virtual scene modulerenders and outputs models of a virtual 3D space. The large-size screendisplay module receives contents of the virtual space and displays themon several LED display screens.

The positioning system comprises an infrared tracking sensor and aninfrared coordinate tracker.

The virtual scene module includes an orthographic camera for shootingvirtual 3D space according to above-mentioned virtual space positiondata. The content of shooting is sent to the large-size screen displaymodule. The orthographic camera utilizes its own principle, instead ofthe perspective to set LED mesh.

The benefits of the present invention include:

The display method of the LED screens is changed from a passive way intoa positive one, which allows images and videos to be displayed indifferent angles according to the changing positions of persons, so asto make real-time change of the displayed contents on the screenaccording to persons' coordinates and enable LED screens to realizereal-time and 3D display. Compared with VR glasses, such a 3D displaymethod of virtual space means that customers don't need to wear heavyand inflexible headsets and annoying wires and cables but can insteadenjoy an easy immersive feeling in the virtual space.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is the composition of the color correction system of the presentinvention;

FIG. 2 is shows the framework of an embodiment of the color correctionsystem;

FIG. 3 shows a sketch map of the prior art using multi-camera infraredpositioning technology;

FIG. 4 shows a sketch map for calculating a position of the cradle headwith current technologies;

FIG. 5 shows a sketch map of the four-sided display space in anembodiment of the present invention implementation cases;

FIG. 6 shows a photo of the multi-viewpoint switched shooting systembased on the infrared positioning system in an embodiment of the presentinvention;

FIG. 7 shows the system framework of the LED 3D image-formation;

FIG. 8 shows an area chart and virtual space, of an embodiment of theLED 3D image formation of the present invention;

FIG. 9 shows a virtual space sketch map of an embodiment of the presentinvention;

FIG. 10 shows a position sketch map of a 3D image formation embodimentof the present invention's system; and

FIGS. 11a-11b show the matching sketch maps of the displayed contents ofthe changed viewpoints of the present invention.

DETAILED DESCRIPTION

Overall unified pictures can be presented by recognizing the shootingposition of the video camera and the parameters related to the camerasand combining real-time color correction software to set the parameters.The output color parameters are related to each screen in real time toadjust the display effect, so as to achieve a unified viewing effectfrom any angle, presenting overally unified pictures whether it be foran audience to watch or for program shooting.

Specifically, a mechanical sensor is adopted with grating coding torecognize the position of the video camera and the camera lensparameters. The related parameters of the video camera are gathered andsent to color correction software. The color correction software setscolor configuration parameters of the video camera when in differentpositions in real-time, and connects and matches the parameters of thevideo camera with the parameters of the sensor. The color correctionsoftware uses related algorithms to deal with parameters in differentpositions, realizing the dynamic match of display effects from differentshooting angles.

This creation offers a multi-screen display color correction method,which includes the following steps:

Step 1, gaining the position coordinate information of the video cameraand images of the multiple screens collected by the above-mentionedvideo camera. The above-mentioned position coordinate information of thevideo camera includes: the coordinate information and camera angleinformation of the video camera.

Step 2, taking one of the above-mentioned multiple screens as thestandard screen and according to the collected images of the multiplescreens conducting color correction on the display contents of the otherscreens, making the display of the images collected by the video cameraon the other screens achieve the same display effect with that of thestandard screen and storing the screen correction parameters conductedto the other screens. For example, include three screens: Screen A,Screen B and Screen C. Screen A is the standard screen. The chromaticaberration values of Screen B and Screen C are adjusted in colorcorrection software, so as to ensure they have the same display effectwith Screen A.

When a camera shoots the display space, due to the structure of LEDscreens, typically the chromatic aberration between two screens occurs.So software is used to adjust the RGB value of the image of each screen.This RGB value is color correction parameters (noted as r1, g1, b1 belowand r2, g2 and b2 below). This RGB value varies depending on therelative position of the camera and each screen. To start, the differentshooting coordinates of up to five cameras are collected (to ensure thecollection of up to five angles) and get the color correction parametersof the same point in the display space respectively. Using this data, afitting function related to the camera position and color correctionparameters (RGB values) is generated. Later in filming, this fittingfunction can be used to adjust the RGB value automatically according tothe camera position, so as to realize the objective of correcting thedisplay color on multiple screens.

The color correction/matching algorithms used is as follows: the cameraposition (x, y, z), the RGB value of Screen B (r1, g1, b1), the RGBvalue of Screen C (r2, g2, b2) wherein the camera position (x, y, z) isan independent variable, and r1, g1, b1, r2, g2, b2 are its dependentvariables respectively. That means:r1=f1(x,y,z)g1=f2(x,y,z)b1=f3(x,y,z)r2=f4(x,y,z)g2=f5(x,y,z)b2=f6(x,y,z)And the f (x, y, z) is used by the data acquired through “least squaremethod”. Step 3, determining the position change times of the positionof the above-mentioned video camera. If the time is less than or equalto N, then change the position of the video camera and return to Step1); if not, skip to Step 4).

Step 4, according to the stored multiple screen color correctionparameters, use the matched curve of algorithm to guarantee the gain ofscreen color correction parameters of the video camera in differentpositions, so as to make images of the several different screenscollected by the video camera in different positions present the sameeffects.

After measuring no less than five points in the same manner, clickcalculation in the software to gain chromatic aberration values fromdifferent angles through the matched curves of the algorithm, so as toguarantee the consistency in the display effects of different screens inthe movement of the video camera.

Step 5, when the video camera is in different positions, conduct colorcorrection on other screens according to the screen color correctionparameters gained by above-mentioned position coordinate information andthe matched curves.

After launching the color correction procedure, the software will readdifferent position coordinates of the video camera and utilize differentchromatic aberration correction parameters for each screen to conductreal-time chromatic aberration correction of the large-size screen.

As shown in FIG. 1, the present invention provides a multi-screendisplay color correction system, which includes: large-size screenserver 10, color correction server 20 and tracking video camera 30 withposition tracking function and three display screens: Screen A, Screen Band Screen C. Video camera 30 collects images of the three displayscreens A, B, C. Color correction server 20 receives the contents sentby large-size screen server 10. The contents of large-size screen 10 arealso the contents displayed on the three display screens A, B, C. Colorcorrection server 20 conducts color correction of the three displayscreens A, B, C according to the contents of the three display screensA, B, C taken by video camera 30 and contents sent by large-size screenserver 10.

Specifically speaking, take one of the display screens as the standard.For example, taking Screen A as the standard screen, conduct the colorcorrection of Screen B and Screen C, with an effort to avoid chromaticaberration in collected images of the three display screens, Screen A,Screen B and Screen C, have differences in the positions and angles ofthe camera lens of video camera 30, leading to different colortemperatures. After correcting the display parameters of Screen B andScreen C, the display contents of Screen A, Screen B and Screen Ccollected by video camera 30 present the same effects, namely noappearance of chromatic aberration, so as to create a vivid simulationenvironment for television programming and film shooting.

FIG. 2 shows the composition framework of the multi-screen display colorcorrection system 100, including color correction server 120 in detail.

Multi-screen display color correction system 100, includes large-sizescreen server 110, video camera 130, large-size display screen 150,color correction server 120 and the multi-screen splicer.

Color correction server 120 includes: image input module 122, positiondata module 124, comprehensive color correction module 126 and imageoutput module 128. Image input module 122 is connected with large-sizescreen server 110. Position data module 124 is connected with videocamera 130. Position data module 124 and image input module 122 areconnected with comprehensive color correction module 126 respectively.Comprehensive color correction module 126 is connected with image outputmodule 128. And image output module 128 is connected to large-sizedisplay screen 150.

Image input module 122 receives screen display contents output fromlarge-size screen server 110. Position data module 124 receives theposition coordinate information output from video camera 130.Comprehensive color correction module 126 receives the video cameraposition coordinate information transmitted by position data module 124,and according to the position information of video camera 130 gainscorresponding screen color correction parameters. Based on theparameters, color correction is conducted on the screen display contentssent by image input module 122. The screen display contents are outputafter correcting the large-size display screen 150 by the image outputmodule.

Comprehensive color correction module 126 can include a storage module,used to store at least five color correction parameters used for colorcorrection and the matched curves of the color correction parameters, soas to be able to gather the screen color correction parameters of thecorresponding multiple display screens based on different positioninformation of the video camera.

Large-size display screen 150 contains multiple display screens, such asScreen A, Screen B and Screen C, making the display effects of themultiple display screens the same. Large-size display screen containsthree LED display screens: Screen A, Screen B and Screen C. The LEDdisplay screens are placed in the left side, right side and the bottomof video camera 130. Video camera 130 contains a video camera trackingsensor. The video camera tracking sensor adopts the mechanical sensorand grating coding to recognize the position and camera lens parametersof video camera 130.

Comprehensive color correction module 126 is able to take advantage ofthe different position information of video camera 130 to gain thescreen color correction parameters of the multiple display screens.Comprehensive color correction module 126 corrects the screen displaycontents of one of the multiple display screens or the multiple displayscreens (such as the display screen B, or the display screen C or thedisplay screen B and C), so as to make the multiple display screens A, Band C have the same display effect.

Before correcting the screen display contents, the display images of theabove-mentioned multiple display screens are collected by video camera130 and one of the display screens is taken as the standard to conductthe color correction of other display screens, thus gaining at leastfive screen color correction parameters, the matched curves of thescreen color correction parameters, and the screen color correctionparameters of the video camera in different positions.

Video camera 130 contains a video camera tracking sensor, which is usedto collect the coordinate information and camera lens angle informationof the video camera. The tracking sensing part of the video camera canutilize a mechanic toothed grating coding sensor such as the one offeredby Shenzhen Oscar Company.

The large-size screen server adopted by the system may be provided byHewlett-Packard Development Company, for example. The video camerautilized may be the video camera with the high definition of Sony. Thetracking sensing part of the video camera can be the mechanic toothedgrating coding sensor offered by Shenzhen Oscar Company, for example.The LED large-size screen may be produced by Leyard, for example. Thelarge-size screen splicer may also be produced by Leyard, for example.An example of the implementation of the present invention is discussedbelow. Three LED screens with different angles (left side, right sideand the ground) are placed; the video camera is placed in a three-meterrocker equipped with a data collection sensor, keeping a 2-meterdistance with the large-size screen; the large-size screen server andthe color correction server are started; the video camera is opened tocheck the shooting pictures; position collection equipment is used tobegin collection of the position of the video camera; the pictures andthe display colors of Screen A, B and C in the video camera are viewedfrom different angles; Screen A is used as the standard and thechromatic aberration values of Screens B and C are adjusted in colorcorrection software to ensure they have the same display effects withScreen A; click “record” in the setting of the software and gain thechromatic aberration parameters of Screen B and C in current positions;after moving to another position, take Screen A as the standard andadjust the chromatic aberration values of Screen B and C to ensure theyhave the same display effects with Screen A; in the same manner, afterat least 5 positions, click the “calculate” in the software to collectchromatic aberration values from different angles by the matched curvesof the algorithm to guarantee the consistency in the display effects ofdifferent screens in the moving video camera; after starting the colorcorrection procedure, the software will recognize the different positioncoordinates of the video camera to use different chromatic aberration tocorrect parameters and conduct real-time correction of the chromaticaberration of the large-size screen.

By recognizing the shooting position of the video camera and parametersrelated to cameras, combining real-time color correction software to setparameters, output color parameters related to each screen are used inreal time to adjust the display effect, so as to achieve a unifiedviewing effect from any angle, presenting overally unified pictureswhether for audiences to watch or for program shooting.

FIGS. 5 and 6 show a multi-viewpoint switched shooting system based onthe infrared positioning system of the present invention. Themulti-viewpoint switched shooting system includes: a multifacetedirregular screen, a video splicer, a 3D rendering server, an infraredpositioning system, a motion-capture computer, a network switch and amultichannel control server.

The video splicer is installed in the back of the multifaceted irregularscreen. The 3D rendering server, the multichannel control server and thenetwork switch are connected with the multifaceted irregular screenrespectively and are used to process the images displayed on themultifaceted irregular screen. The infrared positioning system isinstalled above the multifaceted irregular screen and is connected witha motion-capture computer to track what the video camera is shooting.

The infrared positioning system includes several infrared cameras thatare installed above the space for shooting. Several of the video camerasare placed in the front of the space for shooting and each is equippedwith several infrared markers on the head. The infrared positioningsystem is used to get the position parameters of the infrared markers onthe video camera.

The motion-capture computer is used to correct the difference in valuebetween the infrared markers and the lens of the video camera based onthe position parameters of the infrared markers to get the coordinateinformation of the video camera and transmit it through the networkswitch to the 3D rendering server and the multichannel control server.The 3D rendering server is used to output the display of 3D spaceimages. The multichannel control server is used to match the coordinateinformation of the video camera to the virtual space for usage and matchthe coordinate information of the video camera with the coordinateinformation of the virtual space.

The above-mentioned multifaceted irregular screen includes at least fourLED display screens, which are the front screen, the left screen, theright screen and the ground screen respectively. The multifacetedirregular screens are assembled into a U-shape or a L-shape.

At least 8 infrared cameras are used.

At least 4 3D rendering servers are used and they are independent fromeach other.

A multi-viewpoint switched shooting method based on the infraredpositioning system, includes the following steps:

Step 1, obtaining the position parameters of the infrared markers on allthe video cameras through the infrared positioning system and sendingthe parameters to the motion-capture computer through the network switchIn the infrared positioning system, an infrared camera emits infraredrays while at the same time receives infrared rays reflected by a markpoint on the camera of the infrared rays emitted by the infrared camera.In such a way, the location of the mark point can be confirmed. Theposition of the mark point is close to the camera, and overlaps twocoordinate points by software. The relative position of the two pointswill not change. And the change of the camera position is determinedbased on the change of the position of the mark point.

Step 2, use the motion-tracking software in the motion-capture computerto correct the difference in value between the infrared markers and thelens of the video cameras according to the position parameters of theinfrared markers. Then transmitting the parameter information to the 3Drendering server and the multichannel control server.

Step 3, the multichannel control server projects the parameterinformation of the video camera to the virtual space. Through theadjustment of the image projection algorithms, the parameter informationof the video camera is matched with the parameter information of thevideo camera in the virtual space.

Step 4, the 3D rendering server outputs the display of the 3D spaceimage according to the images taken by the video camera and theparameter information of the video camera in the virtual space.

Step 5, operate the motion-capture computer to switch the video camera.(The location of different cameras are switched to show complete displayspace every time). At the same time, the system switches the angle andposition of virtual scenes, so as to make the displayed 3D space imagematch with the images taken by the video camera. Switch refers toswitching the image shown on the monitor screen from an image filmed byone camera to an image filmed by another camera. The 3D space imagepresents a complete 3D scene through the display of different screens.

As shown by FIG. 6, Leyard's LED screen with a pixel pitch of 1.9 isused, for example, to build up a four-sided display space with the frontscreen of 3×4 meters, left and right screens of 3×4 meters and groundscreen of 4×4 meters. The video splicer can be MVC-2-203 from Leyard,for example. To render the images on these LED screens respectively,four 3D rendering servers were needed, such as HP Z440, 2 meters' awayfrom the front screen, 3 testing video cameras were placed in parallel.The video camera may be Sony's PXW-280, for example. Optitrack systemfrom NaturalPoint, for example, may be used to locate the video camera.This system contains 8 infrared video cameras, distributed in thecorners of the display space.

Through the network switch, a NETGEAR ProSafe, P41 for example, themotion-capture computer collects the position parameters of the infraredmarkers of 3 video cameras by 8 infrared video cameras. TheMotiveTracker and plugins are used, for example, in the computer tocorrect the difference in value between the infrared markers and thelens of the video cameras in the software respectively in order to getthe coordinate information of the video camera and transmit the data tothe four 3D rendering servers and multi-channel controlling serversthrough the network switch. The multi-channel controlling server matchesthe coordinate information of the video cameras to the virtual spacerespectively. Through the mapping algorithm, the coordinate informationof the video cameras and the coordinate information in the virtual spaceare matched and mark the several video cameras in the positioning systemas Point A, Point B and Point C. The motion-capture computer is operatedto switch the video camera. At the same time, the system switches theangle and position of virtual scenes, so as to make the displayed 3Dspace image match with the images taken by the video camera. The 3Dspace image presents a complete 3D scene through the display ofdifferent screens.

One of the technological key points of this invention is a multi-angleswitching shooting and moving method used to shoot multifaceted screen3D space.

Another aspect of the present invention utilizes a positioning system ina real environment, an infrared positioning system, through GPSpositioning, visual positioning, laser positioning, ultrasonicpositioning and other methods matching the coordinates and area of realLED screens to obtain real position coordinates to calculate in thevirtual 3D space. After calculation, the images or videos of thecorresponding coordinates of the virtual 3D space are projected to realcoordinates of LED screens for display. The displayed contents can moveaccording to the person's coordinate in real time. Finally, it makes theLED screen realize the real-time and 3D display. The above-mentionedinfrared positioning system includes an infrared emitter to sendinfrared light to the infrared coordinate tracking sensor, whichacquires the coordinate and area of the objects that are tracked andthen sends the data to the server for processing.

FIG. 7 shows an embodiment of LED 3D image-formation system 700 of thepresent invention. The system includes: server 710, tracking sensor 760,coordinate tracker 770, large-screen splicer 780 and LED display screens750. Server 710 includes three function modules: tracking module 712,large-size display module 714 and virtual scene module 716. Trackingmodule 712 is used to process the person's coordinate position data in areal environment and locate the virtual space after getting the person'scoordinates. Large-size display module 714 is used to display thevirtual space contents, including the displayed contents on the front,left and right, and top and bottom LED screens. Large-size screensplicer 780 is used to match the display contents of the severallarge-size screens 750. Virtual scene module 716 is for rendering andoutputting the model of 3D virtual space.

Referring to FIGS. 8-11, the present invention includes three big spacecoordinates: (1) a Virtual space coordinate 815; (2) a LED screencoordinate in real environment 825 and (3) a person's positioncoordinate 835. The virtual space coordinate 815 is larger than that ofthe LED screen coordinate in a real environment 825, namely the wholeLED screen space 825 being located in the virtual space coordinate 815and the person 835 being located in the space of the LED screen 825.Through the positioning system in the real environment (made up by thetracking sensor and the coordinate tracker), the real coordinates of theLED screens and persons are acquired and later projected to match theposition coordinates in the virtual 3D space environment, and projectthe images or videos of the corresponding coordinates of the virtual 3Dspace to the display of the LED screens, so as to realize the real-timechanging of displayed contents on the LED screens with persons'coordinates and realize the real-time and 3D space display of the LEDscreens.

All of the following are used as examples and no limitations are put onthe specific implementation methods of this invention. Users are wearinga headset equipped with the infrared coordinate tracker. The headset isselected as an example, other methods can be adopted as well, such asthe method of wearing a tracker on the wrist. (As seen in FIGS. 10, 11 aand 11 b). Infrared tracking sensor 1160 is fixed in the realenvironment and emits infrared light to infrared tracker 1170, whichgets the users' position parameters in the real environment and sendsthe collected coordinates to the server. Infrared tracker 1170 and theinfrared tracking sensor 1160 make up the positioning system of theusers' positions. The tracking module is used to get the person'scoordinate position data in the real environment. And according to this,the person's coordinates are set in virtual space. The users' positionsin the virtual space can also be adjusted according to their needs. Forexample, the default position of the user in the virtual space is thecenter, the left or the right side. The users' positions in the virtualspace is then updated in real time according to the changing positionsof the user in the real environment. The large-size display module isused to display the virtual space contents, including the displayedcontents on the front, left and right, and top and bottom LED screens.The large-size screen splicer is used to match the display contents ofthe several large-size screens. The panoramic model of 3D virtual spaceis rendered and output by the virtual scene module, namely including the3D virtual scenes presented from all viewpoints. This 3D virtual spacemodel is finished by the user through a 3D designing software, such as3DMAX, and imported into the server.

The virtual scene module includes setting an orthographic camera tosimulate the real view of a user in the real environment. Theorthographic camera locates the user's position in 3D virtual spaceaccording to the user's position in the real environment to face the LEDmeshes directly in related directions and gaining the scenes of the 3Dvirtual space in the viewpoint of the users. At this time, the positionof the orthographic camera in the 3D virtual space corresponds with theusers' position in the real environment. Then the acquired virtual spacescene contents (including images and/or videos) are output to thelarge-size screen display module, which outputs the virtual space scenecontents to the large-size screen splicer, and finally matched andoutput to the several real LED display screens, so as to realize theupdate of the virtual space contents with the moving of the positions ofactual users.

The server used in the system practice may be the HP Z440, for example.The HTCVIVE-Lighthouse infrared strobe tracking system may be used fortracking. This includes the infrared coordinate tracker and the infraredtracking sensor. The LED large-size screen may be Leyard's screen withP1.9 and a length of 3 meters and a height of 2 meters, for example. Thelarge-size screen splicer may be Leyard's MVC-2-203 and the virtualreal-time rendering engine may be UNITY5.40, for example.

The method of the 3D image formation for a LED display includes thefollowing steps.

Step 1, the first step is completing the production of the 3D modelneeded by the customer in the designing software, such as 3DMAX. Thissoftware model is imported into the virtual real time rendering engine,such as UNITY, to conduct the real-time editing for the 2^(nd) time andthe corresponding plane setting of the area and coordinate of thelarge-size screen in the real environment. The area and coordinates ofthe LED display screen are pre-set in the software by the user, namelyreplacing the real LED display screens by the LED mesh in the 3D modeland using the orthographic camera (designed by the software in the 3Dmodel and replaces the view of real users in 3D models) to face directlyto the LED meshes in related directions. Output the acquired virtualspace image to the large-size screen splicer, which has severalcorresponding LED large-size screens with different resolution ratios toconduct image adjustment and match to finally send to the real screensfor display. The tracking module receives the position parameters of thecoordinate tracker and sends them to the server. The server then makesthe virtual scenes to move and change based on the corresponding data.The orthographic camera outputs images to the LED large-screens in realenvironment to stimulate the virtual space.

The above-mentioned server can be set on the LED screen itself, namelythe LED display screen can include several display screens, and theserver function is equipped in one screen, which serves as the maincontrol screen of the LED display screens. Specifically, this inventionprovides a 3D image-formation screen, which is used to display virtual3D images. This LED display screen includes a main control screen andseveral subordinate screens. The main control screen is made up by thetracking module, the large-size display module, the virtual displaymodule and the large-size screen splicer.

The main control screen is connected with the tracking sensor and thecoordinate tracker respectively.

The above-mentioned coordinate tracker tracks users' positions and sendsthe position parameters to the main control screen.

The main control screen uses the tracking module to deal with users'position parameters, get users' coordinate position data in the realenvironment and converts it into the virtual space position data in thevirtual 3D space. The virtual scene module renders and outputs the modelof the virtual 3D space. The large-size screen display module displaysthe virtual space contents according to the output needs of the virtualspace position data. The large-size screen splicer receives theabove-mentioned virtual space contents and matches them on theabove-mentioned main control screen and the several subordinate displayscreens.

The above cases are examples of the present invention and are not usedto put limitations on the protection coverage of this creation. Anymodifications, equal replacement and advancement made within the spiritand principle of this creation shall be under the protection of theinvention.

The invention claimed is:
 1. A method of maintaining accuracy in a 3Dimage formation system, comprising: correcting a color on multi-screens;the color correction including the following steps: determining positioncoordinate information of a video camera and images of multiple displayscreens taken by the video camera; selecting one of the multiple displayscreens as a standard screen, conducting color correction on displaycontent of the remaining multiple display screens using the imagescollected by the video camera and screen correction parameters, whereinto achieve a same display effect with that of the standard screen;storing the screen correction parameters conducted to the other screens;determining a change time of a position of the video camera, wherein ifa time is less than or equal to N, then changing the position of thevideo camera and repeating the steps above, if the time is greater thanN, moving on to the steps below; and using matched curves of analgorithm from the stored screen correction parameters to guarantee again of screen color correction parameters of the video camera indifferent positions, wherein the images of the multiple display screenscollected by the video camera in different positions present the samedisplay effects, wherein when the video camera is in differentpositions, conducting color correction on the multiple display screensaccording to the screen color correction parameters gained by theposition coordinate information and the matched curves.
 2. The method asrecited in claim 1, wherein the position coordinate information includescoordinate information and camera angle information of the video camera.3. The method as recited in claim 1, wherein the standard screen isselected by selecting one of the multiple screens that faces directlyagainst the position of the video camera.
 4. The method as recited inclaim 1 wherein N equals
 5. 5. The method as recited in claim 1 whereinthe same display effect is a condition that the images of the multipledisplay screens collected by the video camera in different positions andfrom different angles all present a same color and color temperature.